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ABSTRACT 



A system for providing accurate, quantifiable and reproduc- 
ible assessments of an organization's performance based on 
predetermined criteria. The system includes a telecommu- 
nications infrastructure administered by a survey adminis- 
trator. Employees or non-employees of a company or other 
organization to be evaluated log on to the system and answer 
a plurality of questions relating to various aspects of the 
business entity's operations. The information may be 
obtained through Internet communications, through touch- 
tone telephone systems, and in many other ways including 
the use of personal computers, diskettes and email, or even 
manually including the use of pencil and paper where 
penciled-in answers are read by a scanner. The data is stored 
in a database and subsequently analyzed by the survey 
administrator for evaluation and forecasting of the business 
entity's performance. 

24 Claims, 3 Drawing Sheets 
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Do senior leaders guide the company in setting direction and sustaining \ 
effective leadership through the organization? 

O Yes 
• No 

Senior leaders fully guide and sustain effective leadership. 

O Almost Always Disagree 
% Rarely Agree (20%) 
O Seldom Agree (35%) 
O Agree (50%) 
O Mostly Agree (65%) 
O Strongly Agree (80%) 
O Almost Always Agree 
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Scores for a Sample Organization 


Evaluation Categories 


Total Points 


Percent of 
Total Points 


Points Earned 


1. Leadership 


no 


38.2 


42 


2. Strategic Planning 


80 


43.8 


35 


3. Customer and Market Focus 


80 


40 


32 


4. Information and Analysis 


80 


25 


20 


5. Human Resource Focus 


100 


44 


44 


6. Process Management 


100 


36 


36 


7. Business Results 


450 


35.6 


160 


Totals 


1,000 


36.9 


369 
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METHOD FOR EVALUATING CURRENT 
BUSINESS PERFORMANCE 

BACKGROUND OF THE INVENTION 5 

1. Field of the Invention 

The present invention relates generally to a survey system 
and apparatus for evaluating the results. More particularly, it 
relates to a remotely accessible system for the collection of 
employee or non-employee survey responses to quantify 10 
various criteria relating to the operation of an organization. 

2. Description of the Prior Art 

In an increasingly competitive marketplace, business enti- 
ties strive to improve leadership, quality, customer satisfac- 15 
tion and other criteria that directly or indirectly relate to the 
ultimate profitability of the enterprise. Empirical evidence 
has established that business operations that excel in distinct 
and measurable attributes are far more likely to be profitable 
in commerce. As an example, the "Baldrige Index" is made 2Q 
up of publicly traded U.S. companies that have received the 
Malcolm Baldrige National Quality Award during the years 
1988 to 1996. The Secretary of Commerce and the National 
Institute of Standards and Technology ("NIST") were given 
the responsibility, under Public Law 100-107, to develop 25 
and administer the Award with cooperation and financial 
support from the private sector. NIST "invested" a hypo- 
thetical 1,000 in each of the six whole company winners of 
the Baldrige Award. The investments were tracked from the 
first business day of the month following the announcement 30 
of award recipients (or the date they began public trading) to 
Dec. 1, 1997. Adjustments were made for stock splits. 
Another 1,000 was hypothetically invested in the Standard 
and Poor's Index 500 ("S&P") at the same time. NIST found 
that the group of six outperformed the S&P 500 by more 35 
than 2.7 to 1, achieving a 394.5 percent return on investment 
compared to a 146.9 percent return for the S&P 500. 
Therefore, it is desirable for an organization to administer 
periodic assessments of its operations. 

The detailed assessment of a business operation is often a 40 
difficult, expensive, and time-consuming task. Typically, 
upper level management may knowingly or inadvertently 
affect the accurate measure of information gathered for the 
assessment. Outside consultants are often employed to inter- 
view and observe the operation of the enterprise on-site. 45 
However, management may wish to influence the data for a 
number of reasons. Upper level management may be greatly 
affected by the results of the evaluation and may attempt to 
direct the outside consultant only to well performing 
operations, shielding problem areas from discovery. In 50 
addition, the outside consultant may interview subordinate 
employees in the presence of their supervisor. This creates a 
poor environment for gaining candor from the subordinate 
employee on potential areas of improvement that are the 
responsibility of the supervisor. 55 

An evaluation by an outside consultant may require the 
consultant to travel to different geographical locations to 
assess a large operation with continuity. The consultant may 
also require the employee to stop productive work to allow 
time for the assessment interview. It is time-consuming for 60 
an independent consultant to conduct individual employee 
interviews, record the data, and assemble the information 
into a useful form. Furthermore, should a business wish to 
conduct periodic evaluations, there is no guarantee that the 
same consultant will be available. Therefore, the company 65 
cannot be assured that the next independent assessment 
performed will have the same consistency. Nor is there any 
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assurance that secondary evaluations will produce meaning- 
ful results comparable to previous exercises. 

Should the company attempt to apply known economic 
principles to an "in-house" self- assessment, there is the 
potential that more harm than good may come from the 
endeavor. Many organizations begin the process of self- 
assessment with a shallow understanding of the performance 
criteria sought or the optimal method in which to gather the 
information. This self -assessment suffers from an insepa- 
rable relationship between the company's own evaluator and 
the upper management that may exert influence, not only on 
the examinees, but also on the examiner. 

Previous attempts have been made to provide business- 
related surveys such as described in U.S. Pat. No. 5,551,880 
to Bonnstetter et al. (the '880 patent) which is incorporated 
herein by reference. The '880 patent describes a system for 
predicting the potential success of an individual for a 
particular job or task. A survey is conducted wherein the 
employee or potential employee submits information on 
behavioral and value preferences. The information is then 
analyzed and compared against standards for behavior and 
values previously resolved for specific employment. In a 
preferred embodiment of the '880 patent, the behavior and 
value questions are administered through software and 
evaluated via a predetermined algorithm. However, the 
system analyzes the individual employee and not the busi- 
ness entity as a whole. Nor does the '880 patent describe an 
independent party to administer the survey to ensure 
confidentially, honesty, and a full disclosure of the employ- 
ee's perceptions of the business entity. 

Therefore, there is a need in the art for a method and 
apparatus to provide a reliable self- assessment survey pro- 
cess that can be easily administered and scored with the 
accuracy and completeness of a well developed written 
narrative self-assessment. 

There is a further need in the art to integrate existing 
telecommunication technologies to avoid the time and labor 
intensive ordeal of the paper and pencil process or other 
traditional means of survey administration. This would 
allow the database of both individual responses and cumu- 
lative data to be completely external to the organization 
being assessed. 

There is a further need in the art to provide confidentiality 
to organizations and their employees participating in the 
survey. This provides a distinct opportunity to benchmark 
the data to industry peers and to provide the objective 
assurance that all data is accurate and verifiable by a reliable 
survey administrator organization. 

However, in view of the prior art in at the time the present 
invention was made, it was not obvious to those of ordinary 
skill in the pertinent art how the identified needs could be 
fulfilled. 

SUMMARY OF THE INVENTION 

The present invention solves significant problems in the 
art by providing a telecommunications infrastructure that 
supports the efficient, confidential and accurate measure of 
performance criteria relating to the performance of an orga- 
nization. A predetermined set of performance criteria are 
measured by the use of an automated employee and non- 
employee interview system whereby recorded responses are 
stored in a database which is utilized for a quantitative 
evaluation of various aspects of a business enterprise. 

The present invention comprises an array of survey ques- 
tions for evaluating a plurality of predetermined criteria 
relating to the performance of an organization. A survey 
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administrator controls a data-gathering interface means for criteria desired. Furthermore, the operation of the invention 
employees or non-employees of the business to confiden- is highly reproducible and consistent. A baseline assessment 
tially record answers to the survey questions. For each can be repeated over time, allowing the organization to track 
individual employee survey, the data-gathering interface quantifiable results and improvements, 
records the employee's identification, their job classification 5 i t should be noted that the current invention has a wide 
and a business identifier. Pertinent data is also collected for ran ge of applications which may include health care 
each non-employee participating in the survey. The data- accreditation, educational institution assessment, and similar 
gathering interface may comprise an Internet web server, a tasks. For example, the current invention may be applied to 
touch-tone telephone entry system, a paper and pencil sys- a Performance-based Organizational Effectiveness & Effi- 
tem that compiles the answers by scanning them 10 ciency Tracking Program ("POET*) wherein the organiza- 
automatically, and other existing or to-be-developed inter- tional effectiveness and efficiency of child welfare organi- 
face means as well. For example, an additional data- zations are measured in addition to performance indicators 
gathering interface means could be provided in the form of essential to success in a managed care environment, 
a personal computer having appropriate software. A respon- ^ advant of the inveoUon ^ that a assess . 
dent could save his or her survey responses on a diskette and 15 ment of a { corporation may be complet ed more rapidly 
mail the diskette to the survey administrator, or email the and than usi i^pe^t consultants that inter- 
responses to the survey administrator as an attached file^ yiew Qnsite ^ feduced ^ and d it a busifiess 
Tins would enable a respondent to prepare answers and send tQ conduct mQre fr t erformance evaluations that can 
them to the survey administrator while the respondent is on be utihzed lQ & ^ business mQre fitabl 
a night, on the road, or otherwise away from a permanent 20 , _ , . ... 
office environment Another advantage of the invention is that employees of 
_ ' . . i , r , me business are provided with the opportunity to submit 
Tne survey administrator maintains a database of the hones[ and candid aQSWers tQ lentiall aosiltn questions . 

responses to the survey questions and has the ability to Because , he rtunit for external influence to affect the 

archive pas response data. The survey admmistrator then , >s re ^ curtailed the final evaluation of the 

applies a selected formula to the survey data to evaluate the « fa ^ to m ^ ^ ^ 

performance of the organization and produces a quantifiable tions thal ma be subs ' equenl i y improved. 

result based on a plurality of business categories. , f . . . . 

. . , t . . . , . Another advantage of the invention is that the infrastruc- 

In a preferred embodiment, an organization wishing to ^ of , he evaluation fa h ^ hl consistent over dme . ^ 

evaluate the performance of various key aspects of its its , he business emi tQ riodicall measure itself 

operations selects one or more business performance mod- ' • t „ M * „„ „, M „ mo l 

f „ t t r , , : i against previous assessments, 
els. Each business performance model is measured using 

predetermined questions for employees of the business. A Tbss f and other im P ortaQt ob J ects > advantages, and fea- 

survey administrator performs the administration of the tures oi the "vcnUon wiU become clear as this description 

predetermined questions so that confidentially in the inter- proceeds. 

view process insures candid answers to the questions. In one 35 Th e invention accordingly comprises the features of 

embodiment, responses to the questions are gathered utiliz- construction, combination of elements, and arrangement of 

ing an Internet web server and are stored either locally on the P 11 ^ tnat wil1 De exemplified in the description set forth 

server or at a remote location for the survey administrator to hereinafter and the scope of the invention will be indicated 

later compile and analyze. The web server provides the m tne claims, 

opportunity to quickly and efficiently gather information, 40 

particularly where various offices of an organization are BRIEF DESCRIPTION OF THE DRAWINGS 

separated by substantial geographic distance. The web For a fuller understanding of the nature and objects of the 

server also may operate 24 hours per day at relatively low i nve ntion, reference should be made to the following 

cost and permit a plurality of employees to enter in detailed description, taken in connection with the accompa- 

responses at the same time. nying drawings, in which: 

In an alternative embodiment, a telephone system utiliz- FIG. 1 is a diagrammatical view of the survey process 

ing dual-tone multi-frequency ("DTME") input provides a accord ing to the invention; 

means for employees to confidentially record information „ T „ *• i • c iLJ 

relating to the performance of a company. The DTMF entry , n ^? j 2 f is a diagrammatical view of an Internet-based 

has the advantage of not requiring the employee to have a 50 method of retneviD * and recordm S em P lo y ee res P onses; 

machine capable of accessing the web server over the FIG * 3 1S an illustrative example of a survey question 

Internet. Using DTMF entry, the employee logs on using a presented to a user from an Internet web server; 

code provided by the survey administrator. Once within the FIG. 4 is a diagrammatical view of a telephone-based 

telephone system, pre-recorded questions are audibly played 55 method of retrieving and recording employee responses; and 

to the employee. At the conclusion of each question, the FIG. 5 is an illustrative example of an organization's 

employee is prompted to record their response by depressing performance scorecard based on the Malcolm Baldrige 

numerals on the telephone. criteria. 

In another embodiment, non-employees participate in the 

survey as well. However, the questions propounded to 60 DEI AILED DESCRIPTION OF THE 

non-employees may differ from the questions propounded to PREFERRED EMBODIMENT 

employees. Referring initially to FIG. 1, it will there be seen that the 

The data is readily compiled into a useable database organization subject to the survey is denoted by the refer- 

format. At the conclusion of the data gathering stage, the ence number 10 as a whole. Virtually all business entities 

survey administrator can easily analyze the recorded infor- 65 have the following job classifications: executive staff 30, 

mation and produce a report to the business entity evaluating management staff 40, support staff 50, and customer contact 

the entity based on the pre-selected business performance staff 60 as integral employee components which support 
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most all business functions of the enterprise. It is each of 
these job classifications of employees who form the internal 
pool of respondents of the business entities. The login 
procedures for these internal respondents will be discussed 
in more detail later. 5 

In another embodiment of the invention, an external pool 
of respondents is also included. The external pool of respon- 
dents may include consumers, customers or clients of the 
business entity, i.e., those who are not employees or other- 
wise a part of the internal group of respondents. The external 10 
pool of respondents may also be adjusted as desired to 
include members of the board of directors or trustees, 
suppliers or vendors, people who perform outsourced 
operations, competitors, partners, consultants, or any other 
relevant external respondent entity. The login procedure for 15 
these external respondents is similar to the login procedure 
for internal respondents, except that said login procedure 
identifies all members of the external pool of respondents as 
such and by their specific category such as customer, vendor, 
or the like. External respondents may also be queried as to 2 q 
their age, frequency of contacts with the business entity, and 
other information deemed pertinent. 

In many organizations there are one or more consultants 
or holders of other job titles who are on the business 
premises daily and whom might seem to be employees of the 2 s 
organization even though they are technically independent 
contractors. There are other jobs as well where the distinc- 
tion between employee or non-employee may not be easily 
ascertainable. This invention does not depend upon any legal 
analysis of who should be considered internal or external to 30 
the organization as a matter of legal definition because in 
some cases a non-employee such as an outside consultant 
might best be included as a member of the internal pool of 
respondents for survey purposes. Therefore, for convenience 
purposes, and not in any strict legal sense, the members of 35 
the pool of internal respondents are herein deemed employ- 
ees of the organization, and the members of the pool of 
external respondents are herein deemed non -employees of 
the organization. A survey administrator is free to determine 
the respective members of the pools of internal and external 40 
respondents in any way that best serves the purposes of the 
survey without regard to strict legal determinations of the 
legal status of the individual members of either pool. 

The system obtains information from individual employ- 
ees or other respondents regarding their perception of per- 45 
formance criteria existing in the workplace. This is accom- 
plished by utilizing a performance criteria survey. This 
survey is administered and controlled by a survey adminis- 
trator denoted 20. The survey data is obtained by a data- 
gathering interface 70. The data-gathering interface may 50 
comprise a variety of telecommunication mediums which 
may include touch-tone telephone systems, hypertext lan- 
guage format ("HTML") Internet web servers, client-server 
wide area networks ("WANs") or local area networks 
("LANs"). Data-gathering interface 70 acts as a conduit to 55 
receive survey data 90 in response to a plurality of prede- 
termined survey questions. Each question contains several 
important aspects of a business function linked to vital 
performance abilities of the organization. 

The results of the survey are stored in a database 100 for 60 
a baseline analysis of performance criteria and may also be 
used longitudinally in later intervals of time. Prior survey 
data from prior years 110 and 120 may be archived by the 
survey administrator 20 and later utilized as a base-line 
reference to the current evaluation of the organization. Once 65 
a requisite number of respondents or all respondents com- 
plete surveys, an analysis 140 is performed on the survey 
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data 90 by the survey administrator 20 and then communi- 
cated back to the organization 10. 

FIG. 2 provides an overview of an Internet-based data and 
tracking system. A remote user terminal 160 accesses an 
Internet web server 70 through a public switched network 
("PSN") 190. The survey administrator 20 provides the 
organization 10 with a series of codes for the login 210 and 
test administration to begin. Typically, the organization 10 
will choose an alphanumeric password of ten digits or less 
for identification of the organization 10 as a whole. The 
survey administrator 20 records said password and opens it 
for usage in protected area of the website. Next the survey 
administrator 20 assigns a chronological range of numbers 
sufficient to match the total number of respondent employ- 
ees or non-employees elected and eligible to sit for the 
survey. To begin the login 210 process, the respondent 
accesses the world wide web and web server 70 using its 
uniform resource locator ("URL"). The URL may be pro- 
vided as a numerical -based Internet protocol ("IP") address 
or may be accessed by a registered domain name via a 
domain name server ("DNS"). 

It should also be noted that alternatives to web-based 
Internet communications may be employed. The alternatives 
may include 3270 emulation, VT-100, ASCII or separate 
client-server applications. 

In a preferred embodiment, the first web page that appears 
offers an overview of the survey system, including its 
methodology of data collection, processing and aggregation. 
This page also offers the user a link to proceed directly to the 
test. When this option is chosen a new screen appears which 
displays test selection criteria with a sample likert scale. At 
the bottom of this screen, the user is prompted to complete 
three empty fields for password protection entry. The first 
field is for the business ID 220 password that represents a 
unique identification of the organization. The second field is 
for job classification entry 230 that may include coding for 
employee classifications of executive staff 30, management 
staff 40, support staff 50 or direct customer contact staff 60. 
It may also include coding for external respondents, men- 
tioned above, such as customers, suppliers, and the like, and 
there may be a different code, if desired, for each category 
of external respondents. The third field is for a unique 
employee ID 240 that relates to the preset numbers matching 
the number of employees as indicated above. 

Once this information is entered a confirmation process 
250 is triggered that compares this keyed-in information to 
the information on server 70. If a match is detected the 
server then determines as at 260 if the user has previously 
initiated the survey. If the user has previously initiated the 
survey, a cache 280 record of the user's answers is accessed 
and the user is only presented with the survey questions that 
were not answered. 

If a mismatch 170 in passwords is detected during the 
login process 210, an error screen is displayed and the user 
has the ability to reenter the three identification codes 220, 
230, and 240. It should be noted that additional fields may 
be added to the login process which may include length of 
employment, demographics and other data. Alternatively, a 
single login ID may be utilized which individually identifies 
the employee's organization, job classification and personal 
identification. 

Upon successful login 210 and confirmation 250, the 
survey is commenced as at 270 with a series of questions 
presented in HTML format. Under each question is a form 
input section coded in HTML which may comprise a check 
box, radio button, or edit box. Alternatively, other input 
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means compatible with the user's web browser may be 
utilized including ActiveX or Java. Responses to the input 
means are stored in a database 100 that may reside on the 
web server or at a remote location. The questions and answer 
input forms may be configured on a single scrollable page or 5 
on individual HTML pages joined by hyperlinks. 

FIG. 3 discloses a sample question screen as viewed 
through an Internet web browser comprising a plurality of 
input sections to each survey question topic. Users are 
prompted to respond to each of the questions in one or more 10 
components. In the following example, there are three 
components. Component A queries whether the respondent 
believes a specific performance criteria exists in their work- 
place. Component B queries the respondent to weigh their 
level of agreement or disagreement to a functional extrapo- 15 
lation of component A. Component C asks the respondent to 
check one of many methods how such a performance criteria 
could be improved. 

In the example of FIG. 3 two question components are 
presented. Question 1 proposes a query on the perceived 20 
leadership skills in the company and provides for a Boolean 
response. The first query is followed up by another query 
that requests a quantified response with seven possible 
entries. In the preferred operation of the Internet system, 
none of the responses are selected when the user first 25 
downloads the page to their web browser. Each response 
must be completed for the "next question button" to become 
available for next question cueing and for the responses of 
that user to be filed in the database 100. Should the user 
attempt to skip over the questions without responding, the 30 
internal controls validate whether a response has been 
entered and repeat the same page to the user until a com- 
pleted response is received by the web server. In a preferred 
embodiment, JavaScript is utilized to validate completed 
responses. The question-to-question responses are tempo- 35 
rarily stored in a cache 250 until all survey questions are 
completed and the data is then committed to database 100 
for analysis. An interrupt feature is available to all respon- 
dents who are unable to finish the survey in one sitting. If the 
respondent stops and logs off, he or she may restart the 40 
survey at any later time, by simply re-entering the password, 
job classification code and test ID. This procedure opens an 
incomplete survey and restores it to the last position or next 
available question from data stored in cache 250. When all 
questions are completed and a "save last answer question" 45 
button is selected, the survey is flagged as done 260 for final 
database 100 entry. Analysis 140 is available when all users 
complete their survey. Data from this part of the survey 
helps the organization evaluate the current level of organi- 
zational consistency about perceptions of effectiveness and 50 
efficiency between and among different levels of staff, across 
all operations, between staff members, and between and 
among the pool of external respondents as well, if any. As 
improvements are made the organization can re-visit the 
assessment in future years to track scores in key areas, to 55 
monitor improvements and to encourage greater perfor- 
mance. A score card system also provides a context for an 
organization's scores to be compared to the respective 
scores of other organizations. 

In the touch-tone telephone embodiment shown in FIG. 4, 60 
the questions are audibly prerecorded for playback. The 
employee or non-employee is given a telephone number to 
call with a set of access codes which may comprise a 
business ID, job classification and employee ID, if 
applicable, as described in connection with FIG. 2. The user 65 
dials into the system from a standard touch- tone phone 300. 
The user is prompted for each identifier which is entered by 
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pressing the numbers of the touch-tone telephone. When a 
key is pressed on a touch-tone telephone, a DTMF signal is 
emitted which is correlated to the numeral on the keypad by 
a system processor. These systems are widely used in 
automatic call distributor ("ACD") systems utilized by call 
centers and by banks and financial institutions to provide 
automated account information. In the example, after the 
user is authenticated prerecorded segments 320 of the survey 
are played back to the user. As an option, the user may press 
the STAR (*) or POUND (#) key on their telephone to repeat 
the last playback segment. After a survey question is audibly 
presented to the user, the user then may answer the question 
depending on the response options using a DTMF input 330. 
For a Boolean question which requires a "yes" or "no" 
response, the user may be prompted to press "1" for "yes" 
or "2" for "no." Alternatively, should the survey question 
require a quantitative response, the user may be prompted to 
enter a range of numbers on the telephone keypad. For 
example, should the user "Almost Always Disagree," they 
would depress "1." If the user "Rarely Agrees," they would 
depress "2." The scale may continue so that should the user 
"Almost Always Agrees," they would depress "7." However, 
it should be noted that the telephonic entry is not limited to 
single digits. For example, should the user be required to 
indicate a percentage numeral of 55%, the entry may be 
executed by depressing in sequence: "5," "5," and then the 
"#" key. Should the user fail to respond to the system with 
a DTMF signal 330 within a predetermined period of time, 
the system may replay as at 340 the previous question 
proposed. 

Both the Internet-based and telephonically-based system 
preferably time-stamp the entries wherein each time a 
response is posted to the web server or entered into the 
telephone system, a corresponding record of the date and 
time is also stored in the database. This may be utilized for 
an assessment of the amount of time needed on average to 
complete the evaluation and when each employee submitted 
their responses to the survey. 

FIG. 5 illustrates a diagrammatical overview of an orga- 
nization's total quality output scorecard based on the Mal- 
colm Baldrige criteria. However, other performance mea- 
surements and formulas may be applied within the scope of 
this invention. The purposes of the Baldrige Assessment are 
to promote an understanding of the requisites for perfor- 
mance achievement and competitiveness for business enti- 
ties and to promote the sharing of information on successful 
business practices. The Baldrige Assessment consists of 
seven criteria: (1) leadership, (2) strategic planning, (3) 
customer and market focus, (4) information and analysis, (5) 
human resource development and management, (6) process 
management, and (7) business results. 

Leadership is evaluated by the ability of senior executives 
to guide the company and how the company addresses it 
responsibilities. Strategic planning examines how the com- 
pany sets strategic directions and how it determines key 
action plans. Customer and market focus examines how the 
company determines the requirements and expectations of 
customers and markets. Information analysis examines the 
management, effective use, and analysis of data and infor- 
mation to support key company processes and the compa- 
ny's performance management system. Human resource 
development and management examines how the company 
enables its workforce to develop its full potential and how 
the workforce is aligned with the company' objectives. 
Process management examines aspects of how key 
production, delivery and support processes are designed, 
managed, and improved. Business results are examined by 
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the company's performance and improvement in its key 
business areas: customer satisfaction, financial and market- 
place performance, human resources, supplier and partner 
performance, and operation performance. Business results 
are also evaluated by the performance of the company 5 
relative to its competitors. 

In a preferred embodiment, each question has both a 
yes/no and weighted score component so that data can be 
accurately tallied with sophisticated analysis and report 
generation. Instead of using a simple 1-5 Likert scale which 10 
is subject to a wide range of interpretation, the current 
invention uses quantified behavioral descriptors, dramati- 
cally increasing score reliability. As previously illustrated in 
FIG. 3 a written question is followed by a quantified 
(percentage-tagged) behavioral description. This form of 15 
test construction yields an increase in the consistency and 
accuracy of ratings as well as greater coherence of linkages 
between business processes. The method of utilizing a 
written question followed by a quantified description is well 
known in the health care industry. In health care it was 20 
discovered that by obtaining input this way from a cross 
section of functions and levels throughout the organization, 
a performance profile can be developed that not only iden- 
tifies strengths and areas for improvement, but deployment 
gaps as well. This is something that written narrative assess- 25 
ments alone do not effectively provide. Accurate survey 
data, based on behavioral follow-up questions, can be used 
to compare or benchmark organizations within and among 
industries, and can also support longitudinal performance 
studies. 30 

The system represents an item by item, group by group, 
performance improvement digest that allows management 
the ability to recognize the linkage between performance of 
key processes to quality improvement outcomes (including 
the future allocation of resources to achieve such outcomes). 35 

The system does this by compiling each respondent's raw 
data and generating a written digest of an organization's 
strengths and opportunities for improvement. Data is gath- 
ered and assembled to provide an organization with a 4Q 
detailed analysis in the form of a printed report. Responses 
in the database 100 are analyzed as at 140 and reported in 
several ways. The database table is constructed to include 
vertical columns formatted with mathematical formulas rep- 
resenting yes or no response totals, raw Likert averages, 45 
adjusted Likert averages and Organizational Success Ratios 
("OSRs"). Dividing the adjusted Likert by the raw Likert 
produces an OSR. This number can be represented in the 
form of a ratio or quotient and generally as a two place 
decimal ranging from 0.01 to a perfect 1.00. The OSR is 5Q 
used to profile the system's highest and lowest scores in 
performance improvement, representing current quality 
strengths and weaknesses in the business entity. 

The OSR data collected from the employees and non- 
employees, if any, is consistent across the organization. The 55 
results can present a clear profile of strengths and current 
weaknesses from which the business entity can plan appro- 
priate actions. Each OSR is then converted to a Baldrigc 
equivalency formula that converts the two place decimal 
point to a percentage. The percentage number is then applied 60 
line by line to re-create the points available in each section 
found in the sample table. The "maximum" points for each 
category of the Malcolm Baldrige National Quality Award is 
listed with the actual score. 

As illustrated in FIG. 5, a maximum of 450 points for the 65 
"business results" category may be obtained, even if the 
sample organization returned only 160 points. To convert 
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each business category result to a ratio, the total points 
achieved in each category is divided by the overall maxi- 
mum number of points possible for the category. Thus, 160 
divided by 450 equals 0.356 or 35.6 percent. By converting 
each business category measurement to a ratio, the organi- 
zation may easily obtain a quantifiable assessment of its 
performance. 

It will be seen that the objects set forth above, and those 
made apparent from the foregoing description, are efficiently 
attained. Since certain changes may be made in the above 
construction without departing from the scope of the 
invention, it is intended that all matters contained in the 
foregoing description or shown in the accompanying draw- 
ings shall be interpreted as illustrative and not in a limiting 
sense. 

It is also to be understood that the following claims are 
intended to cover all of the generic and specific features of 
the invention herein described, and all statements of the 
scope of the invention which, as a matter of language, might 
be said to fall therebetween. 

Now that the invention has been described, 

What is claimed is: 

1. A method reliant upon anonymous responses to survey 
questions for quantifying predetermined criteria relating to 
the performance of an organization, comprising the steps of: 

providing a library of performance assessment surveys 
where each performance assessment survey includes a 
plurality of survey questions; 

providing a plurality of pools of respondents; 

providing an automated survey administrator; 

enabling said automated survey administrator to custom- 
ize a survey by preselecting a performance assessment 
survey from said library of performance assessment 
surveys; 

further enabling said automated survey administrator to 
customize a survey by preselecting a pool of respon- 
dents from said plurality of pools of respondents; 

further enabling said automated survey administrator to 
conduct multiple surveys simultaneously where each 
survey is directed to a different pool of respondents; 

authenticating each member of a pool of respondents 
before enabling each member to answer said survey 
questions; 

acquiring information about each respondent that identi- 
fies what knowledge about the organization is held by 
said respondent; 

presenting survey questions to each respondent that call 
for information held by said respondent, and selecting 
said survey questions from a plurality of said perfor- 
mance assessment surveys if the knowledge of said 
respondent indicates an ability to answer said survey 
questions; 

providing a data-gathering interface connected to a global 
computer network by means of a remote web server, 
thereby enabling said pool of respondents to individu- 
ally record answers to said survey questions in private 
and with anonymity; 

providing a database containing answers submitted by 
said pool of respondents; 

providing a predetermined formula for evaluating said 
answers; 

producing a quantified result, using said predetermined 
formula, based on said predetermined performance 
criteria; 
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storing said quantified result in said database; 8. The method of claim 1, further comprising the step of 

retrieving from said database quantified results previously enabling said automated survey administrator to further 

stored therein from earlier performance assessment customize a survey by parsing said performance assessment 

surveys for comparison purposes; survevs t0 create a surve y that incorporates preselected 

. . t c t . . r . • , . r 5 features of said performance assessment surveys. 

said step of retrieving fierier including the step of 9 . The method of claim 1, mrther comprising the step of 

enabling said automated survey administrator to pre- enabH &M automated s administrator to further 

select quantified results from said earlier performance customize a survey by preselecting a pool of respondents 

assessment surveys; from a plurality of pools of respondents. 

wherein said automated survey administrator preselects 10. The method of claim 1, wherein the step of providing 

which performance assessment surveys will be utilized, a plurality of pools of respondents includes the steps of 

based upon the respective knowledge held by respon- providing a pool of executives of said organization, a pool 

dents in said pools of respondents; 0 f managers of said organization, a pool of shareholders of 

wherein said automated survey administrator preselects said organization, a pool of employees of said organization, 

what pool of respondents will be enabled to participate a pool of customers of said organization, a pool of vendors 

in the survey; of said organization, a pool of government regulators that 

wherein said automated survey administrator conducts enforce laws relating to said organization, and a pool of 

plural surveys simultaneously, directing a different sur- members of the general public so that said human survey 

vey at each pool of respondents and wherein said organizer may further customize a survey by selecting any 

automated survey administrator directs a plurality of 20 combinations of pools or respondents, 

surveys to any preselected pool of respondents; 11. The method of claim 1, further comprising the step of 

wherein said automated survey administrator preselects enabling members of said preselected pool of respondents to 

what comparisons of quantified rests will be made; anonymously provide answers to said survey questions by 

wherein the respondents are enabled to answer said ques- ^putting answers into a web page under the control of said 

tions in private and in anonymity; and 25 ^ administrator. 

. , . _ , , _ . , 12. The method of claim 1, further comprising the step of 

wherein the quantified results of said performance assess- enabH members of said lected f of t0 

ment are available immediately after the performance , j j 

, y v anonymously provide answers to said survey questions 

assessment in commenced^ mrQ h a computing device such as a ersonal 

2. The method of claim 1, wherein the step of providing CQ and b a hand . held mternet ^ 
a remote web server to record said answers further com- 30 ^ ^ rf ^ autQmated suryey administrator 
pnses the steps ol, 13 ^ method of claim ^ farther comprising tne slep Q f 

enabling individual members of said predetermined pool enabling members of said preselected pool of respondents to 

of respondents to access a website server over said anonymously provide answers to said survey questions by 

global computer network; manually completing an answer sheet. 

displaying predetermined survey questions in the form of 14. The method of claim 1, further comprising the steps 

a question and answer sheet at a website hosted by said 0 f : 

website server; enabling said automated survey administrator to preselect 

enabling said individual members of said predetermined a p00 i 0 f respondents from multiple categories of 

pool of respondents to answer said survey questions by ^ respondents, including executives, managers, 

inputting information into said question and answer employees, customers, shareholders, vendors, govern- 

sheet; and ment regulators, and members of the general public, 

inputting said information into said database. 15. The method of claim 1, further comprising the step of 

3. The method of claim 2, wherein said global computer enabling said automated survey administrator to authenti- 
network is accessed by a wired computing device such as a 45 C ate each member of a preselected pool of respondents 
personal computer. before allowing such member to answer said survey ques- 

4. The method of claim 2, wherein said global computer tions. 

network is accessed by a wireless hand-held appliance. 16. The method of claim 1, further comprising the step of 

5. The method of claim 1, further comprising the step of enabling the automated survey administrator to educate each 
providing a Boolean response to said survey questions 5Q respondent about predetermined aspects of the performance 
followed by a quantified response to the same survey topic. assessment method so that each respondent may provide 

6. The method of claim 1, further comprising the step of better educated answers to the survey questions propounded 
including the Malcolm Baldrige National Quality Award by the automated survey administrator. 

Criteria as defined by the United States National Institute of 17. The method of claim 16, wherein the step of enabling 

Standards and Technology as one of said performance 55 the automated survey administrator to educate each respon- 

assessment surveys. dent about predetermined aspects of the performance assess- 

7. The method of claim 1, further comprising the step of me nt method further includes the steps of: 

resolving said quantified result by: providing pre-survey interactive training that educates 

scoring the survey responses relative to a predetermined individual respondents about each question and the 

score based on determination on the importance of each 60 entire survey; 

performance criteria; providing a pre-survey on-line tutorial that is automati- 

correlating the survey responses for performance criteria cally scored and formatted to pre-qualify the respon- 

categories into distinct arrays of data; dent to sit for the survey; and 

adding numerical values to said arrays of data represent- providing plural assistance options for answering concur- 

ing separate performance criteria categories; and 6 5 rent survey questions accurately in-real time so that 

dividing said numerical values by a maximum theoretical respondents will better understand the survey questions 

value of said performance criteria category. and thus provide more meaningful answers. 
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18. The method of claim 1, further comprising the step of 
linking preselected self-assessment criteria to a performance 
assessment survey and employing statistical inference to 
estimate future expected values for management-specified 
performance indicators. 

19. The method of claim 1, further comprising the step of 
inter-rating the respondents to identify those who give 
answers that may be disregarded. 

20. The method of claim 19, further comprising the step 
of classifying according to a predetermined statistical deter- 
mination the reliability of responses relative to underlying 
biases that are evaluated using conjoint analyses of 
responses to key questions. 

21. The method of claim 1, further comprising the steps 
of linking together the tasks of survey building, survey 
taking, and analysis of results. 

22. The method of claim 21, further comprising the steps 
of tutoring the organization in the choice of survey questions 
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and analysis methods, developing a dialog with respondents 
to coach them in proper responses and assist them in the 
completion of the survey, conducting a real-time analysis of 
individual survey participant responses and accumulating 
said responses in a respondent data base. 

23. The method of claim 1, further comprising the step of 
evaluating Boolean responses from respondents using a suite 
of multi-variate statistical methods to determine cross- 
participant survey trends by category of response and to 
establish predictive opportunities for performance improve- 
ment based on evaluated process capability ratios. 

24. The method of claim 1, further comprising the step of 
identifying improvement areas based on process capability 
indices and performance gaps that are generated by com- 
parison of survey responses to benchmark survey responses 
in a performance data base. 
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